Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[develop] Add custom munge key rotation script #2453

Merged
merged 4 commits into from
Sep 18, 2023

Conversation

hehe7318
Copy link
Contributor

@hehe7318 hehe7318 commented Sep 11, 2023

Description of changes

  • Create a custom munge key rotation script for customer to manually fetch munge key from secrets manager in case the secret ARN doesn't change but the value changes

Relate Pull Request

New sample YAML configuration

DevSettings:
  SlurmSettings:
    MungeKeySecretArn: Str

@hehe7318 hehe7318 requested review from a team as code owners September 11, 2023 18:57
@codecov
Copy link

codecov bot commented Sep 11, 2023

Codecov Report

Patch and project coverage have no change.

Comparison is base (4c5b1e9) 76.06% compared to head (13a0246) 76.06%.

Additional details and impacted files
@@           Coverage Diff            @@
##           develop    #2453   +/-   ##
========================================
  Coverage    76.06%   76.06%           
========================================
  Files           13       13           
  Lines         1876     1876           
========================================
  Hits          1427     1427           
  Misses         449      449           
Flag Coverage Δ
unittests 76.06% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@hehe7318 hehe7318 changed the title Munge key rotation [develop] Add custom munge key rotation script Sep 11, 2023
Copy link
Contributor

@EddyMM EddyMM left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to maintain a temporary copy of the existing munge key and use it to restore the system back to a consistent state in case an error occurs after replacing the key (e.g. munge service fails to restart)? Or maybe fallback to a key generated by Munge? Let's sync on this.

exit 1
fi

# Remove current munge key if exists
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See comment in the update PR: 2452/files#r1325705545

I think we can simply override the existing key without explicitly removing it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have safely removing these codes.


# Enable and restart munge service
systemctl enable munge
echo "Start to Restart munge service"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: echo "Restarting munge service"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done!

systemctl enable munge
echo "Start to Restart munge service"
systemctl restart munge || { sleep 10; systemctl restart munge; } || { sleep 10; systemctl restart munge; } || { sleep 10; systemctl restart munge; } || { sleep 10; systemctl restart munge; }
echo "Restart munge service completed"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: echo "Restarted munge service"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also done for echo "Sharing munge key" and echo "Shared munge key"

# Enable and restart munge service
systemctl enable munge
echo "Start to Restart munge service"
systemctl restart munge || { sleep 10; systemctl restart munge; } || { sleep 10; systemctl restart munge; } || { sleep 10; systemctl restart munge; } || { sleep 10; systemctl restart munge; }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason for the multiple sleep commands? Is it to wait for the service to restart? I think systemctl restart is a synchronous operation (unless otherwise specified: --no-block).

Did you notice asynchronous behaviour while running the command?

Also after restarting the munge service we can check if it's running systemctl --quiet is-active munge.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

def enable_munge_service
  service "munge" do
    supports restart: true
    action %i(enable start)
    retries 5
    retry_delay 10
  end
end

I add the retry codes because I mentioned the enable_munge_service contained retry steps. So I added it in case there is something I don't really understand in the munge service.

But yes, I think we can remove it.

Also I added the checking commands.

EddyMM
EddyMM previously approved these changes Sep 15, 2023
@hehe7318 hehe7318 merged commit b32d548 into aws:develop Sep 18, 2023
hgreebe pushed a commit to hgreebe/aws-parallelcluster-cookbook that referenced this pull request Nov 13, 2023
* Munge key rotation

* Refined the rotation script
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants